Web Survey Bibliography
An overlooked source of survey measurement error is the misclassification of open-ended responses. We report on efforts to understand the misclassification of occupation descriptions in the Current Population Survey (CPS). We first analyzed a set of occupation descriptions (n=32,362) reported by CPS respondents and entered by interviewers; each description was classified by two independent coders. One factor that was strongly related to reliability was the length of the occupation description: contrary to our intuitions, longer descriptions were less reliably coded than shorter ones. We followed this with an experimental study in which we constructed occupation descriptions (n=800) to vary on key features (e.g. specific terms led to low or high reliability in study one); these descriptions were again double-coded. Finally, we asked coders to classify 100 experimental descriptions and report on their thinking while doing so. One practice that was evident in coders' verbal reports is that they use informal rules that may improve reliability but have no bearing on the accuracy of coding.Based on these data, we propose possible interventions to improve intercoder agreement.
Homepage (abstract)/(presentation)
Web survey bibliography - Conference on Optimal Coding of Open-Ended Survey Data, 2008 (8)
- CAQDAS, Secondary Analysis and the Coding of Survey Data; 2008; Fielding, N.
- Machines that Learn how to Code Open-Ended Survey Data: Underlying Principles, Experimental Data, and...; 2008; Sebastiani, F.
- Computer coding of 1992 ANES Like/Dislike and MIP responses; 2008; Fan, D. P.
- CATA (Computer Aided Text Analysis) Options for the Coding of Open-Ended Survey Data; 2008; Skalski, P.
- Classifying Open Occupation Descriptions in the Current Population Survey; 2008; Conrad, F. G., Couper, M. P.
- Coding Responses Generated by Open-Ended Questions: Meaning Matching or Meaning Inference?; 2008; Potter, Ji.
- Open-ended questions and text analysis; 2008; Popping, R.
- Coding Verbal Data - What to Optimize?; 2008; Krippendorff, K.